Meta-metrics for the Accuracy of Software Project Estimation

Software project estimation for such items as Size, Effort, Cost, Delivery time, Reliability and Risk, is a fundamental skill for software engineers. In order to improve estimation, there is a need for measurement of the measurements, that is, a meta-metric for the process. This paper compares three existing metrics and provides an example of conversions between them. It then extends one (DeMarco's Estimating Quality Factor) to recognise and reward rapid convergence to an accurate figure during a project. A second metric is defined to provide a lower bound on error for multiple initial estimates. The paper concludes with a worked example as an illustration of the metric's properties. 1. Background One of the major problems of Software Engineering is the lack of confidence in estimates of project factors such as Size, Effort, Cost, Delivery time, Reliability and Risk, even though these are of fundamental interest to the client. Poor estimates lead to poor plans and inadequate planning is a basic cause of failure with software projects. Despite substantial developments in recent years in the field of Software Metrics, industry's ability to predict basic project parameters is low and the proportion of systems delivered significantly overdue remains unacceptably high. A previous paper (Woodings, 1995) considered a taxonomy of Software Metrics with particular reference to giving greater visibility to measures of process improvement. This paper considers the issue of meta-metrics for the accuracy of measurement of project parameters in general (and development effort in particular) and the need for specialised metrics which may be monitored to provide evidence of organisational improvement. The need for meta-metrics was emphasised in some recent research (Lederer, 1998), which indicated little improvement in estimation practices due to new models and techniques. It asserted: "Only one managerial practice the use of the estimate in performance evaluations of software managers and professionals presages greater accuracy. By implication, the research suggests somewhat ironically that the most effective approach to improve estimating accuracy may be to make estimators, developers and managers more accountable for the estimate even though it may be impossible to direct them explicitly on how to produce a more accurate one." However, the more managers are held to their promises, the more they will be tempted to adjust other project factors in order to look good. Predictions become self-fulfilling prophesies. Thus, there is a need to have in place a framework of meta-metrics to monitor, guide and provide feedback to organisations on their project management.